The impact of training on rater variability
نویسنده
چکیده
In the five years, 1993 to 1998, total Commonwealth Government spending on education fell from 4.9 to 4.4 per cent of Gross Domestic Product. Australian universities have responded to this changed funding environment though the increased use of casual teaching staff. The aim of this study was to develop, implement and evaluate a short, cost-effective training package designed to improve the rating performance of causal teaching staff. The pre and post training performance of a group of raters was measured using the Partial Credit Model, an extension of the Rasch model.
منابع مشابه
Reduction of Inter-Rater and Intra-Rater Variability in Psoriasis Area and Severity Index Assessment by Photographic Training.
BACKGROUND Severity grading is important for the assessment of psoriasis treatment efficacy. This is most commonly achieved by using the psoriasis area and severity index (PASI), a subjective tool with inherent inter-rater and intra-rater variability. PASI-naive dermatologists require training to properly conduct a PASI assessment. OBJECTIVE In the present study, we aimed to investigate wheth...
متن کاملOn Rater Agreement and Rater Training
This paper first analyzed two studies on rater factors and rating criteria to raise the problem of rater agreement. After that the author reveals the causes of discrepencies in rating administration by discussing rater variability and rater bias. The author argues that rater bias can not be eliminated completely, we can only reduce the error to a cetain degree by training raters. The study on r...
متن کاملA Study of Raters’ Behavior in Scoring L2 Speaking Performance: Using Rater Discussion as a Training Tool
The studies conducted so far on the effectiveness of resolution methods including the discussion method in resolving discrepancies in rating have yielded mixed results. What is left unnoticed in the literature is the potential of discussion to be used as a training tool rather than a resolution method. The present study addresses this research gap by exploring the data coming from rating behavi...
متن کاملStudents’ Oral Assessment Considering Various Task Dimensions and Difficulty Factors
This study investigated students’ oral performance ability accounting for various oral analytical factors including fluency, lexical and structural complexity and accuracy with each subcategory. Accordingly, 20 raters scored the oral performances produced by 200 students and a quantitative design using a MANOVA test was used to investigate students’ score differences of various levels of langua...
متن کاملRaters’ Perception and Expertise in Evaluating Second Language Compositions
The consideration of rater training is very important in construct validation of a writing test because it is through training that raters are adapted to the use of students’ writing ability instead of their own criteria for assessing compositions (Charney, 1984). However, although training has been discussed in the literature of writing assessment, there is little research regarding raters’ pe...
متن کامل